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ABSTRACT 

Informal science education institutions have been identified as critical participants in 
helping students succeed in science by working in collaboration with school systems across the 
country. The results of one such collaboration, the Urban Advantage (UA) program found that 
participation in UA improved student achievement, on average, by 0.6 standard deviations on the 
8 th grade New York State Intermediate Level Science exam. However, while some UA schools 
performed exceedingly well, others performed well below expectations. In the current study, we 
explore the heterogeneity in the results and look deeper into what may be the cause of this 
variation, focusing closely on the school-level factors that may help or hinder success. After 
identifying a set of high and low-performing UA schools, we use qualitative research methods to 
uncover the ways in which successful schools are implementing UA, as well as other school- 
level factors that may influence the degree to which the school is able to benefit from the UA 
program. Identifying these best practices across different school contexts may help UA program 
staff develop strategies to support UA schools having more limited success. Additionally, 
findings from this study may aid UA staff in the school selection process, for example choosing 
schools they previously may not have based on their school-level characteristics. 
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I. INTRODUCTION 


Today, no work may be more important or timely than re-envisioning the approaches, 
leadership, and methodologies for educating U.S. children in science. Within the formal 
education sector, researchers have pointed to the importance of supporting students’ college- and 
career-readiness by more closely aligning K-12 education standards with the knowledge and 
skills they will need to succeed in introductory college-level science courses (The College Board 
2009). At the same time, informal science education (ISE) institutions are seeking to support the 
science efforts of school systems that have increasingly focused their attention on 
reading/language arts and math. These ISE institutions have been identified as critical 
participants in helping students succeed in science, “premised on the notion that their emphasis 
on phenomena-rich, learning-driven interactions with science resonates with the notion of 
inquiry underlying K-12 science education reform” (National Research Council 2009). 

Earlier work done by two of the authors on one such collaboration — the Urban 
Advantage (UA) program — found that participation in UA improved student achievement by 0.6 
standard deviations on the 8 th grade New York State Intermediate Level Science exam 
(Weinstein, Whitesell & Schwartz, 2012). These results, however, reflect the average effect of 
attending a UA school and mask significant variation between schools. 

While some UA schools performed exceedingly well, others performed well below 
expectations. In the current study, we explore the heterogeneity in the results and look deeper 
into what may be the cause of this variation, focusing closely on the school-level factors that 
may help or hinder success. We identify a set of high and low-performing UA schools in order 
to determine what these schools may be doing differently. After identifying these two sets of 
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schools, we use qualitative research methods to uncover the ways in which successful schools 
are implementing Urban Advantage, as well as other school-level factors that may influence the 
degree to which the school is able to benefit from the UA program. 

Identifying these best practices across different school contexts may help UA program 
staff develop strategies to support UA schools having more limited success. Additionally, 
findings from this study may aid UA staff in the school selection process, for example choosing 
schools they previously may not have based on their school-level characteristics. 

This paper is organized as follows: Section II describes the UA program while section III 
provides a review of the literature. Section IV describes the data, quantitative methods and 
results and how we chose our case study schools. Section V presents our findings, section VI 
contains the discussion and section VII has our conclusion. 

D. BACKGROUND 

Launched in 2004 as a collaboration between eight NYC ISE institutions (the American 
Museum of Natural History, Brooklyn Botanic Garden, New York Botanical Garden, New York 
Hall of Science, Queens Botanical Garden, Staten Island Zoological Society, and the Wildlife 
Conservation Society’s Bronx Zoo and New York Aquarium) and the New York City 
Department of Education (NYCDOE), the UA program provides both students and teachers with 
opportunities to engage in authentic science practice. Taking part in the scientific process - 
designing and conducting investigations in which they pose scientifically oriented questions, 
prioritize evidence, and develop logical explanations - is a prerequisite to understanding science 
(The College Board 2009; National Research Council 2005; National Research Council 2007). 
Grounded in the learning goals defined in the New York State Learning Standards for 
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Mathematics, Science, and Technology, UA is focused on supporting teachers to help their 6th, 
7th, and 8 th grade students carry out long-term scientific investigations, including the “science 
exit projects” New York City (NYC) 8th graders are expected to complete before progressing to 
9th grade. 

The name “Urban Advantage” reflects the partners’ belief that it is an advantage to live in 
an urban setting with so many science-rich cultural institutions and nature facilities. UA differs 
fundamentally from traditional museum-to-school collaborations as it provides a hybrid model 
for civic engagement where the resources of institutions are selected, designed, and shaped to 
align specifically to the science curriculum of NYC’s middle schools. 

UA provides 48 hours of professional development for new teachers and ten hours each 
year for continuing teachers. 1 The professional development model is designed using an 
immersion into inquiry strategy (Loucks-Horsley et al. 2009). Professional development 
emphasizes authentic hands-on learning experiences in science, the nature of scientific work, 
specific science topics, and the essential features of inquiry in the form of long-term 
investigations (National Research Council 2000). After choosing a UA partner institution to 
participate in professional development with, teachers learn how to plan effective field trips, 
embed resources in instruction, use UA-provided equipment and resources, and teach students 
the components of experimental design as well as how to develop scientific explanations based 
on claims, evidence, and reasoning. Teachers conduct their own scientific investigations, 
learning first-hand what it means to “do science,” which is consistent with the teacher-as-learner 
model of professional development (Thompson and Zeuli 1999). Additionally, over the course 
of its existence, the UA program has developed a variety of classroom tools and school and 

1 These are teachers who are new to the UA program, not new to the profession of teaching. 
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family related resources to support outside-of the-classroom science learning. Now in existence 
for ten years, UA serves 525 teachers and almost 53,000 students and their families in 173 
middle schools — about 32percent of NYC's public middle schools in the 2013-14 school year — 
across all five boroughs. With support from the AMNH, informal science education institutions 
and school districts in other cities have taken steps to implement programs based upon the UA 
framework in NYC, with a UA program launched in Denver in the 2010-11 school year. 

In its early years, teachers and schools self-selected into UA. Over time, the program has 
developed a more rigorous protocol for accepting both teachers and schools. This is partly due to 
increased demand and partly due to budget reductions as a result of the fiscal constraints 
experienced by the New York City Council and NYC Department of Education, which fund the 
program. Rather than expand to provide the program to more schools, program staff have opted 
to grow within already participating schools, opening the program to 6 th -grade teachers and 
hoping to attract more than one teacher per grade. Additionally, UA staff have developed more 
professional development offerings for continuing teachers and the balance of participants has 
shifted from new to continuing teachers over time. These workshops are designed to focus in 
greater depth on specific content related to the science exit project process and provide 
opportunities for experiences teachers to examine students work and thinking. These sessions 
are open only to teachers who had already participated in new teacher professional development. 
To help ensure ongoing participation in the UA program, attendance at continuing teacher 
workshops are required for teachers to continue to receive resources and classroom materials 
provided by the program (Short et al. 2012). 
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III. REVIEW OF THE LITERATURE 


The Need for Collaboration 

Collaboration between schools and informal science education organizations is important 
for a number of reasons. Participation in informal science education has been found to play a 
role in student’s long-term career decisions by further engaging students, encouraging authentic 
inquiry, developing academic knowledge and understandings, developing self-efficacy in 
science, decreasing external barriers and increasing supports, and exposing students to STEM 
careers, particularly among women and those from minority and low-income communities 
(Dorsen, Carlson, and Goodyear 2006; Clewell and Darke 2000; Fadigan and Hammrich 2004). 

Formal and informal institutions contribute differently to students’ science learning in 
part because of structural differences. Schools are not designed for ongoing, authentic science 
investigations, as they must operate within the constraints of a school setting and generally have 
fewer science-specific resources. As such, it is unrealistic to encourage formal education to 
model itself after informal education; rather, a comprehensive science education is achieved 
through collaborations among many different types of institutions, both formal and informal 
(Adams, Gupta, and DeFelice 2012). Rosser (1997) describes collaborations between in-school 
and out-of-school learning experiences as a “two-pronged approach to learning.” Many science 
education groups agree: the National Research Council, National Science Teachers Association, 
the National Science Board, and the Institute of Museum and Fibrary Science all assert that 
collaboration between schools and informal science institutions is important. 

Relationships between formal and informal science education institutions take various 
forms, all of which seek to combine complementary aspects of formal and informal settings, 
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maximizing their benefits (Phillips, Finkelstein, and Wever-Frerichs 2007; Adams, Gupta, and 
DeFelice 2012). Beyond single-day field trips, most ISEs have both teacher and student 
programs. For teachers, ISEs frequently provide teacher residency programs, research 
opportunities, and professional development; for students, ISEs typically feature family outreach 
programs, camp-ins, activity kits, and various activities and materials (Astor-Jack, Balcerzak, 
and McCallie 2010; Hein 2001; Hofstein and Rosenfeld 1996; Inverness Research Associates 
1996; Ramey-Gassert, Walberg, and Walberg 1994; Kisiel, 2010). In the Center for Informal 
Learning and Schools’ survey of 345 informal science institutions, 73 percent reported providing 
“support in the way of programmes, workshops, materials, curricula, etc. for districts, schools, 
teachers, or students in the broad area of science education besides a one-day field trip” (Phillips, 
Finkelstein, and Wever-Frerichs 2007). 

Unfortunately, these ISE-based resources tend to be underutilized. Fifty-three percent of 
the informal institutions responding to the Center for Informal Learning and Schools’ survey 
reported that their programs could handle more participants than they currently serve, while only 
24 percent indicate they have to turn away potential participants due to capacity constraints 
(Phillips, Finkelstein, and Wever-Frerichs 2007). Collaborations between formal and informal 
education institutions may help increase the utilization of existing informal science resources. 

In fact, the key goal of partnerships between formal and informal settings is to support 
cross-contextual learning. As articulated by Voss (201 1), rich cross-contextual learning 
experiences are more than simple field trips; in addition to students traveling to informal 
institutions, informal educators may travel to schools to provide informal- style learning 
opportunities in the formal school context. Cross-contextual learning events tend to be most 
successful when they truly span both formal and informal settings. For example, it is beneficial 
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when school-based and informal educators work together to plan learning experiences and when 
formal educators introduce concepts prior to informal learning, facilitate students’ reflection of 
informal learning experiences, and assess the learning that takes place in informal settings (Voss 
2011 ). 

Fidelity of Program Implementation 

How well a program is implemented is related to how well a program does, especially 
when the program is being implemented in multiple sites, with different contexts, settings, and 
players. Therefore, we reviewed selected studies on factors of successful program 
implementation and features associated with positive outcomes. 

There are several different aspects of program implementation that influence intended 
outcomes. Dane and Schneider (1998), identify five aspects of fidelity of implementation that 
have been commonly measured: (1) adherence (i.e., extent to which specified program 
components are delivered as prescribed); (2) dose or exposure (i.e., amount of program content 
participants receive); (3) quality of program delivery (i.e., extent to which providers approach 
ideals with respect to delivering program content and process); (4) participant responsiveness 
(i.e., extent to which participants are engaged); and (5) program differentiation (i.e., uniqueness 
of features of program that can be reliably distinguished from others). Possibly the most integral 
piece of implementation is fidelity is the integrity to which the intervention corresponds to that 
which was originally intended. In outcome research, fidelity is the confirmation that the 
manipulation of the independent variable occurred as planned (Moncher & Prinz, 1991). 
Carrying out the intervention as intended, therefore, is major driving factor to producing 
expected outcomes. 
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How the various aspects of implementation interact with each other may also be relevant 
to understanding the process. For example, there can be a tension between fidelity and adaption, 
as success relies on delivering the program as intended, however, it also may require some 
degree of flexibility in particular settings (Dusenbury, et ah, 2003). Dane & Schneider (1998) 
note that several authors (Bauman et al., 1991; Meyer et al., 1993, for example) have suggested 
that program modifications to accommodate local needs are necessary and acceptable, provided 
that the critical features of a program are delivered as planned. 

Multiple studies also examine contextual factors associated with successfully 
implementing programs. A comprehensive review of literature on implementation of this 
particular type of program finds several critical factors to implementation success. These 
include: teamwork and composition; top management support; creating an organization specific 
business plan and vision; business process reengineering with minimum customization; project 
management; monitoring and evaluating performance; effective communication; and software 
development, testing and troubleshooting. (Fui-Hoon Nah, et al., 2001). Among these many 
factors, top management support, which involves a commitment from management and 
willingness to allocate resources to the implementation effort, is key. Parr et al. (1999) find that 
management support, appropriate skills of implementation team and commitment to change are 
the strongest contextual factors influencing success. 

Durlak and DuPre (2008) examine studies affecting the implementation process of 
prevention programs. Overall, 23 relevant factors were identified and support the conclusion that 
contextual factors must be considered when innovations are implemented in real world settings. 
One important factor highlighted is compatibility, which includes contextual appropriateness, fit, 
match, and congruence between the program and the setting. Additionally, the community 
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context in which a program will be conducted also likely influences implementation. One 
important community factor identified by the authors is politics. 

Program implementation in various settings is complex, as organizations, even of the 
same type, are often dynamic and diverse. The literature we have reviewed on program 
implementation outlines various aspects of the implementation process and also discusses 
specific contextual factors that influence the success of the program. It appears that both factors 
matter and that both may influence program outcomes. 

IV. DATA 

Our quantitative analysis draws on a rich student-level longitudinal database for NYC 
public schools and students from 2003-04 to 2009-10. Student data include socio-demographic 
characteristics (age, gender, race/ethnicity, birthplace), educational needs (special education, 
limited English proficient, eligibility for free lunch), and standardized test scores (statewide 
English and math tests in grades 3-8 and science tests in grades 4 and 8). Our sample includes 
eighth-graders from 2004-2010, for a total of more than 400,000 student-year observations. 

We measure short-term outcomes using the eighth-grade Intermediate Level Science 
(ILS) exam. New York State requires that all eighth-grade students take the ILS test, which 
consists of approximately 80 questions in three sections: multiple choice, open-ended, and 
performance-based questions. The test covers three standards: scientific inquiry, living 
environment, and physical setting. We measure student performance on the ILS with a 
standardized score (“z-score”), which is a measure of relative performance and is standardized 
across students within a grade to have a mean of 0 and a standard deviation of 1. Students 
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performing above (below) average relative to other students in their grade have positive 
(negative) z-scores. 

The measure of UA program participation we use is at the school level. We define a 
school as being a UA school if at least one teacher in the school is in the UA program; thus, 
different schools will have different concentrations of UA teachers within the school. We never 
“turn off’ the treatment at the school level, as once teachers have been exposed to the UA 
program, it is impossible to retract the treatment. Thus, once a school becomes a UA school, it is 
always a UA school for the purposes of our analysis. Note, however, that we only include UA 
schools that are in the program for at least two years, since the program is unlikely to be fully 
implemented until year two. 

Identifying High and Low Performing UA Schools 

To identify relatively high- and low-performing UA schools, we draw on a school fixed 
effects model we used in our previous analyses to estimate the impact of UA on science 
achievement. Results from this model (Table 1) show that students at UA schools perform, on 
average, approximately .06 standard deviations higher on the eighth-grade ILS exam than 

9 

students at non-UA schools." From this model we predict school effects, which capture each 
school’s unique contribution to student science achievement after controlling for UA 
participation, student characteristics, students’ pre-existing performance trajectories, and year 
effects. Each school has a single school effect, which reflects its average contribution to student 
science achievement across all years, above and beyond what is explained by UA participation 
and student characteristics. 


2 To take into account the fact that it takes time to train teachers and implement the program, schools that are only in 
UA for only one year are excluded from the analysis. When also excluding schools that are only in UA for two or 
three years, point estimates are similar. 
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The variation in school effects reflects the fact that some schools are very successful in 
raising student science test scores in ways that are not reflected in our model (potentially through 
a school-wide focus on math and science, extracurricular activities, or a more supportive school 
culture), while science achievement at other schools may be inhibited by factors not included in 
our model (potentially through low attendance or a chaotic school environment). Positive school 
effects indicate that schools contribute positively to a student’s science achievement, above and 
beyond what is explained by the other variables in the model, notably UA participation and 
student characteristics; negative school effects, on the other hand, indicate there are some 
unmeasured school factors that hinder students’ science achievement. 

As shown in Table 2, the average school effect at schools that are never in UA is close to 
zero (0.007), meaning that on average, non-UA schools have little impact on science test scores 
beyond what is explained by student characteristics. Schools that are UA for at least two years 
have an average school effect of -0.027, and schools that are ever in UA for at least four years 
have an average school effect of -0.014. This indicates that on average, UA schools do not 
contribute positively to the science achievement of their students above and beyond what is 
explained by UA participation and student characteristics. This makes us more confident that our 
estimates of the impact of UA are not simply reflecting differential selection into the program - 
“better” schools are not disproportionately selecting into UA. Rather, because students in UA 
schools outperform other students on science exams despite negative school effects, we are 
confident that our estimates reflect true program effects, not other aspects of school quality. 

Note also that UA schools have less variation in school effects, with higher minimum 
effects and lower maximum effects than schools that are never in UA. The ten UA schools with 
the highest and lowest school effects (across all years) are displayed in Tables 3. We describe 
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our methods for selecting our case study schools in the following section, but note that we 
included two schools from the list of the ten UA schools with highest school effects (Galileo 
School and Salem Junior High School) in our qualitative sample. 

Selecting Case Study Schools 

Using the results above from the above analysis, we identified 64 UA schools with 
positive school effects and 89 UA schools with negative effects. Working with UA staff, we 
developed a set of criteria for choosing schools to participate in the case studies. First, the 
schools had to be active in the UA program in the 2012-13 school year in order to facilitate their 
participation. Second, the schools had to have relatively high, positive school effects. Third, we 
sought to include schools from across all five boroughs. Finally, we attempted to include schools 
that varied both in terms of student socioeconomic status and demographic characteristics. We 
ultimately recruited a mix of schools that ranged from 31% to 97% eligible for free or reduced- 
price lunch; demographically mixed; for example, at two schools more than 80% of the student 
body is white, at one school nearly two-thirds of the students are Asian, and at several schools 
the student body is more mixed. We ultimately recruited seven highly-performing schools. 

Only 29 of the lower performing schools were still active UA schools in 2012-13. 
Because of the difficulty of recruiting these schools, we could not rely on the other criteria for 
selection. After contacting a number of schools, we were finally able to recruit two that were 
willing to participate in the study. 

Having identified our nine schools for qualitative inquiry, we utilized the qualitative 
methods of interviews and classroom observations to investigate the implementation of the 
program and identify best practices. The interview protocol consisted of questions that were 
developed based on our previous research reports that articulate potential moderators for the 
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impact of UA. 3,4,5 We attempt to uncover previously unobserved characteristics of each school 
we study to identify how each of the UA strategies work within each school that have enabled 
them to improve the science achievement of their students. Our conceptual framework draws a 
connection between the various components of the UA program, school-level factors that 
influence the implementation of UA practices, and student science outcomes. The impact of 
program components themselves may be mediated or moderated by school-level factors, such as 
teacher collaboration, administrative leadership, and the role of the parent coordinator. For 
example, UA materials may be utilized more successfully in schools where teachers have time 
and space to collaborate with each other and align instruction across grades. Similarly, access to 
institutions is a key component of the UA program, but schools without sufficient support from 
the administration or help from the parent coordinator may find it difficult to organize field trips 
and consequently under-utilize this resource. 

Unlike the school characteristics we included in our quantitative models, such as percent 
free or reduced-price lunch (which come from administrative data), more nuanced 
characteristics, such as collaboration among teachers, concentration of UA teachers, and 
administrative support may provide valuable insight into what makes UA successful in certain 
settings. 

At each of the schools, we interviewed the principal or assistant principal who oversees 
the science department, the parent coordinator, and both UA and non-UA science teachers. We 
also sought to conduct observations in science classrooms of both UA and non-UA teachers. In 


3 Meryle Weinstein, Robyn, Sperling, and Nazar, Lila (2011). Urban Advantage: The First Six Years (2005-2010). New York: 
Institute for Education and Social Policy. 

4 Meryle Weinstein, Robyn Sperling, and Einilyn Ruble (2011). UA Leadership Institute Final Report. New York: Institute for 
Education and Social Policy. 

5 Robyn Sperling and Meryle Weinstein (2012). Analysis of Lead Teacher Teaching Practices Urban Advantage. New York: 
Institute for Education and Social Policy. 
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total we conducted 34 interviews: one principal, four assistant principals, three parent 
coordinators, 20 UA teachers and six non-UA teachers. Unfortunately, at four schools we were 
able to interview only one teacher. We also observed 19 classrooms, in which members of our 
research team took notes individually and later used the Reformed Teaching Observation 
Protocol (RTOP) to quantify the behaviors observed in the classroom. 

Table 4 shows the demographic characteristics and academic performance of the nine 
case study schools. Schools were located in four of NYC’s five boroughs: Manhattan, Brooklyn, 
Queens, and Staten Island. All schools had been in UA for at least two years, and some had been 
in the program for as many as five years. Seven of the schools in our case studies are traditional 
middle schools, which serve grades six through eight, while the other two schools serve grades 
six through twelve. Three of the schools are relatively small middle schools (150-350 students in 
grades 6-8), two are medium sized (about 600 students in grades 6-8), while four are quite large 
(900-1800 students in grades 6-8). 

The schools represent a wide variety of different racial compositions, with most having 
high concentrations of minority students, and one school is an an all-girls school. There is also 
considerable variation in terms of student poverty, ranging from 31% to 97% free or reduced- 
price lunch students. In terms of educational needs, the percent English language learners ranged 
from 1% to 38%, and the percent qualifying for special education services ranged from 5% to 
21%. Our two schools with negative school effects had average test scores below the citwide 
average and passing rates ranging from 25% to 44%, depending on subject; among our more 
successful schools, academic performance was strong but varied significantly. 

V. FINDINGS 
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Our main findings from the qualitative analyses are organized below: they are school 
culture and structure, teacher collaboration, concentration of UA teachers, administrative 
support, UA vouchers, and school participation in other programs. 

School Culture and Structure 

A major theme to emerge from this study is that UA typically functions best when it can 
build off of existing resources and ethos within schools. The culture at the school and in the 
science department must be conducive to a collaborative, inquiry-based approach; additionally, 
structural factors (e.g., grade span, size) affect the extent to which UA can be successfully 
implemented in schools. 

At one grade 6-12 school, for example, there is a positive school culture, and UA has 
helped to foster a cohesive vision for the school. One teacher explained, “It's a family here. The 
staff is all very close. [...] One [visitor] said, this is the most fun teacher meeting I've ever been 
because you guys all clearly like each other.” The assistant principal at this school noted, “I do 
think them getting Urban Advantaged trickles down,” suggesting that UA has been a contributor 
to establishing a coherent vision in the science department. Teachers agree that the school has a 
cohesive vision for what science should look like at the school: “The school definitely has a 
shared mission overall. [. . .] The middle school mission is really more to try to get them excited 
about [science] and interested in it. My biggest goal, I guess, is to make the kids like science and 
not think of it as something that’s scary.” 

This unifying vision around inquiry was observed in all classes (both middle and high 
school levels). Students were frequently engaged in hands-on activities and working together in 
groups, with teachers serving as resources. Additionally, teachers frequently challenged students 
to engage in a problem independently before they were guided to the “correct” answer. 
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If only a small group of teachers or administrators are invested in the program, UA is not 
likely to be successful. Similarly, as described below (see the “Collaboration” section), UA will 
not automatically turn a science department into a collaborative team. UA promotes teacher 
collaboration and is most successful in collaborative environments, but if teachers are not 
receptive and if the school does not have the necessary structures in place (e.g., common meeting 
times, such as grade or department level meetings, shared meeting space), simply having some 
teachers in the program is not likely to change the status quo. 

This theme of UA expanding on existing positive attributes is also seen with regard to the 
use of resources in the school. Perhaps the best example of this is voucher use. If schools have a 
positive view of fieldtrips and have necessary structures in place (e.g., substitute teachers, 
coverage plans, chaperone systems), UA vouchers will encourage teachers to take students on 
field trips by reducing expenses and helping with transportation. UA vouchers are not likely to 
be enough to induce teachers to take class trips in the absence of supportive administrators and 
systems. 

Several aspects of school structure can have an influence on how UA is implemented and 
on how successful it is in different schools. First, grade span and school size work in conjunction 
to determine how many students of what ages attend each school; cohort size in particular affects 
how many different teachers there are for each grade and subject area, and this ultimately has 
ramifications for both student and teacher experiences. Additionally, teacher turnover is a 
school-level factor that can influence the extent to which UA can be successfully implemented. 

UA is a program for science teachers and students in grades six through eight, and these 
grades exist in different types of schools; in our case studies, we focus on traditional middle 
schools and middle -high schools. Each configuration has benefits and drawbacks. For example, 
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the middle-high schools have the benefit of fewer school changes for students and a more 
seamless transition from eighth to ninth grade; furthermore, collaboration between middle and 
high school teachers may help to prepare middle school students more appropriately for high 
school classes. At one 6-12 school, for example, a high school teacher explained how they 
envision middle school years to be about building excitement and interest in science, and high 
school years to be about hands-on field research: 

“And in high school, in particular, the science mission is very much to get students 
involved in primary research and field work. [...] The middle school mission is really 
more to try to get them excited about it and interested in it so they want to do that type of 
stuff when they’re in high school because there aren’t many field work research 
opportunities for middle school kids because they’re too little.” 

One potential drawback of middle-high schools, however, is that there may be less of an explicit 
focus on the unique needs of middle school-aged students than there is at a traditional middle 
school. We find that school grade span and size can influence the extent to which it is possible 
for teachers to collaborate with each other, as described in the “Collaboration” section below. 

Teacher turnover rates also seem to be related to a school’s ability to implement UA 
successfully. This pattern could exist for several reasons. First, if there is turnover within the 
science department, then it will be difficult to maintain a cohesive vision among teachers, and 
alignment across different classes may suffer. Further, if UA teachers leave the school and 
reduce the concentration of UA teachers on staff, then it may be especially difficult for UA to 
gain traction or maintain momentum within the school. Even across departments, schools that 
experience high levels of teacher turnover may not be able to foster a culture of collaboration. 
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Furthermore, high turnover may be indicative of other school-level issues that make it difficult to 
successfully implement programs such as UA at the school. 

Most schools in this case study had teacher turnover rates between 5% and 15% in 2009- 
10. 6 It is not unusual to see significant jumps or dips in teacher turnover rates from year to year, 
due to random events (e.g., multiple teachers moving out of state) or specific events (e.g., the 
arrival of a new principal). Thus, it is important to consider trends in turnover rates. Because we 
do not know whether the teachers who left the schools were science teachers, turnover rates are 
best viewed as a reflection of general teacher churn within the school. Despite the fact that 
turnover rates are not science specific, it is notable that two of the three schools in our study that 
have consistently high turnover rates are not performing as well as expected in science. 

Teacher Collaboration 

UA can be a uniting force to increase collaboration among teachers, but this seems to 
work best in departments that already have a culture of sharing and structures in place to make 
collaboration possible. Teachers must have the opportunity to meet together frequently, both in 
department and grade-level meetings, and there also must be time and space for informal co- 
planning. The schools where we observed the greatest degree of collaboration had multiple UA 
teachers in the same grade as well as different grades, planned frequent department and grade- 
level meetings, and had a common space that facilitated frequent but informal teacher-initiated 
check-ins. 

Two schools we studied reported having very high levels of collaboration among science 
teachers. At both schools, the administration is very supportive of collaboration and provides 


6 Teacher turnover rates for 2009-10 are calculating by dividing the number of teachers in 2009-10 who do not return to the 
school in 2010-1 1 by the total number of teachers in the school in 2009-10. This fraction is multiplied by 100 to become a 
percent. 
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time and space for teachers to meet together. At one school, teachers frequently eat lunch 
together and work during their prep periods in an informal science office; they also meet 
regularly in department meetings and visit each other’s classrooms. The science department 
exhibits strong coherence across different classrooms and grade levels. We observed students in 
different grades using the same process for their long-term science investigations, with many of 
the same graphic organizers (including the DSET, IDD, and other school- specific handouts), 
intermediate deadlines, rubrics, etc. The teachers confirmed this in their interviews, as they 
explained how as a science team they have worked together to establish a common vision and a 
common plan for executing these long-term investigations. The teachers asserted that this type of 
collaboration is easier for them now that most of the teachers in the school are in UA. 

All schools we spoke with described some form of department or grade-level meetings, 
though the frequency and usefulness of these meetings varied across schools. For example, three 
schools reported having monthly science department meetings but having little opportunity for 
real collaboration, as those meetings are used for administrative purposes or other more general 
task-oriented work. At one school, for example, science teachers work together on common 
assessments during department meetings, but teachers do not always know what is happening in 
other classrooms. While no schools reported having frequent (more than once monthly) 
department meetings, teachers at several schools reported meeting more frequently with their 
grade-level teams. These meetings seem to be more focused on administrative tasks or grade- 
level logistics as opposed to providing a space for collaboration about the curriculum or teaching 
practices. 


7 The DSET and IDD are UA designed tools for use with classroom based scientific investigations 
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One major constraint to having more collaboration among science teachers is the lack of 
a common free period for teachers to meet with each other. One teacher explained, “I mean, I 
would love to [plan together]. It's for a lack of, you know, the time and our prep periods are 
always different and things like that.” Without a designated free period to meet, teachers are 
often left to check in with each other quite briefly and spontaneously. For example, at one 
school, the science teachers who have neighboring classrooms try to speak to each other between 
classes or drop in on each other’s classes to have a brief check-in: 

“The three science teachers in the sixth grade. . . our classrooms are right next door to 
each other. So if we have a couple of minutes in between or during our class, we’ll peek 
in on each other and try to collaborate as much as possible so that we are on the same 
page, and we share ideas.” 

At one school, the assistant principal has recognized that common planning time would help to 
facilitate deeper co-planning and collaboration among teachers, and so next year the school is 
planning to implement departmental common planning periods. 

Another structural aspect of schools that can have a major impact on teacher 
collaboration is the number of teachers who teach the same course. One issue that arose in the 
small schools is that when there is only one teacher for each grade, it is difficult to collaborate 
with colleagues, even if they are also in UA. Teachers seem to naturally work more with other 
teachers who teach the same material, as they are teaching the same standards and potentially the 
same curricula, and teachers seemed generally more aware of what was happening in the 
classrooms of teachers who were in their grade level. At one school, for example, a teacher 
explained: 
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“Because we only have one teacher in each division. . . there's not a whole lot of co- 
planning that goes on except ki nd of overall skills-based stuff. We look over each other's 
assessments or projects, and give each other feedback sometimes. But there's not a whole 
lot of, like, unit planning together or planning shared assessments because we teach 
different classes.” 

Teachers at several schools who felt they were not able to collaborate much with other science 
teachers in their schools expressed that a major benefit of UA is the opportunity to collaborate 
with other science teachers who are teaching the same grade level or content area. 

Concentration of UA Teachers 

The concentration of UA teachers within the science department is an important factor 
for how UA is implemented in schools and the extent to which UA teachers are able to integrate 
UA practices throughout the science department. UA is best viewed as a school-based reform, so 
schools where only a small percentage of science teachers are in UA are not set up structurally to 
reap the greatest benefits of the program. 

In seven of the nine schools in our case study, the majority of science teachers are in UA, 
and in two schools all science teachers are in the program. In all schools, at least two teachers are 
in UA, so teachers have at least one other person at their school who has UA training and who at 
least occasionally uses UA materials. We did not find any systematic differences in the 
concentration of UA teachers in the schools with positive and negative school effects. This may 
be because a variety of other school-level factors (e.g., time and structures for teacher 
collaboration, use of vouchers, support from the administration) vary by school and are also 
critical to UA implementation and to a school’s success in science overall. 
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Many of the science teachers we spoke with indicated that having several teachers in UA 
is helpful, both for departmental cohesion and student experiences. One teacher explained, “The 
fact that so many more teachers have been on board now, now we’re really working jointly, and 
just following much of the same path”; another teacher described how that with “everybody 
using the same philosophy and the same tools, it’s a much more cohesive experience for the 
kids.” Similarly, another teacher explained how repeated exposure to UA can streamline 
instruction: “So it’s something they’ve already seen, so there’s not a whole lot of introduction 
that’s really needed. It’s more of a refresher course.” 

While having all science teachers in UA would allow the greatest degree of UA 
coherence and continuity both within and across grade levels, this is not always possible. In 
addition to the percent of the science staff that participates in UA, the size of the department 
overall and also how the teachers are distributed within the department may matter. When 
several teachers of the same grade (e.g., sixth-grade science) are in UA, those teachers are able to 
align grade-specific curricula, projects, and rubrics to UA principles; when UA teachers are 
distributed across different grades, teachers can vertically align their curricula and projects so 
that students use the same tools multiple years in a row. 

However, even when teachers work together and use the same tools, teachers do not 
always have time to differentiate the tools to be appropriate for each grade level, which speaks to 
the importance of structures that enable collaboration. One teacher explained: “We don’t have 
very much time to meet as a science team and, like, sort through, for example, like, the 
complexity of the DSET, like, what it should look like in sixth grade, what it should look like in 
seventh. I think we kind of do it on our own.” 

Administrative Support 
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We consider both general administrative support of science teachers as well as the 
administration’s support of the UA program. First, in terms of general administrative support, all 
the teachers we interviewed believe that their administration is supportive, but this is manifested 
differently in different schools. For example, teachers at most schools explained that if they ever 
needed anything for their classrooms, their administrators would try to help them to purchase 
materials. Teachers also described administrative help with student issues, lesson plans, and 
everyday teaching responsibilities as well as feeling that the door is always open for frequent 
informal interactions. 

One key way teachers feel supported by administrators is by being given autonomy to 
make their own decisions. For example, at one school, teachers indicated that administrators are 
very supportive and trust the science department to do what is best for their students. The 
assistant principal explained that instead of interfering or micromanaging, her approach is to hire 
smart people and then give them the freedom to do what they think is best: “I think [it’s about] 
getting really smart people and providing them the autonomy to do what they want to do. [. . .] 
just giving teachers full license and support.” 

At another school where science teachers are relatively autonomous, some science 
teachers felt the science department was overlooked. One teacher explained, “They’re pretty 
supportive. [. . .] just trust us, like, with whatever we say pretty much. And the downside of it is 
there’s not much time in the school spent on science.” Another teacher agreed: “I thi nk the 
principal is trying to support the science department. But I feel, you know, just the way the 
school is set up and I think this is true of most city schools, you know, the emphasis is on ELA 
and math and science gets kind of pushed to the side.” 
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Teachers in another school also feel that administrators are focused on math and ELA at 


the expense of science instruction. One teacher explained how she frequently participates in 
ELA-themed professional development, and much of the support she receives is focused on 
writing across the curriculum instead of teaching science content. She feels limited in what she 
can do, and she also believes that her administration - despite knowing she is struggling to cover 
all the science standards - keeps pushing more ELA-oriented responsibilities onto her and her 
class (e.g., long-term essays). 

At most schools we studied, teachers indicated that UA validates science teachers and 
provides science-specific support that they do not have access to elsewhere, either in their 
schools or through outside professional development. One common theme is that UA provides 
some of the best - and often the only - science professional development available to teachers. 
Even teachers who express displeasure with the logistics around professional development (e.g., 
missing school, giving up time on the weekend, commuting from far away) often continue to 
participate in the program because of the quality of the professional development. 

Several teachers at different schools explained that they joined UA specifically for the 
professional development, as they have not received science-specific professional development 
in the past. At another school, an alumnae teacher with more than twenty years of teaching 
experience continues to participate in the UA professional development because she finds it 
helps support her teaching practice, especially with the investigation process and aligning her 
curriculum to the Common Core Learning Standards. 

In terms of the administration’s support of UA, most teachers reported that their 
administrators are supportive of the program. Many teachers cited the fact that their principals or 
assistant principals attend the UA principals’ breakfasts, while others highlighted how the 
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administration promotes science events, such as family field trips, science fairs, and Family 
Science Night Out} These events are often organized by the parent coordinator, assistant 
principal, and teachers. At other schools that have not typically held these types of events, 
administrators believed that they could and potentially should work harder to reach out to parents 
and provide these types of programs. 

Another way principals are supportive of the UA program is by strongly encouraging 
their teachers to join UA, as opposed to letting it spread naturally through the department. For 
example, one Assistant Principal (AP) asserted, “We definitely push it. It’s something that we 
want them to do.” Some administrators believe in the benefits of UA specifically, whereas others 
simply hke their teachers to be involved in external programs and to leverage as many resources 
as possible. At one unique school, several teachers believe the principal pushed them to join UA 
entirely because he wanted to receive more UA vouchers and then use them for field trips 
associated with another program (described in the “Other Programs” section). Teachers in other 
schools have not felt a push from the administration to get involved; rather, it has been more of a 
bottom-up process, with one teacher joining the program and then encouraging others to join. 
Schools where teachers became involved through a bottom-up process often have proactive 
teachers in collaborative science departments. 

UA Vouchers 

All of the schools we studied reported using the UA vouchers, although to different 

degrees and for different purposes. Three of the schools in this case study make school field trips 

an integral component of the entire school. One school features an integrated curriculum, where 

teachers plan collaborative projects for students and take them on at least three field trips a year 

8 Family Science Night Out is a science-fair type event with student demonstrations and other 
science activities, and it is encouraged and supported by the UA program. 
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At this school, UA vouchers are typically used to help facilitate these field trips; in fact, teachers 
indicated that one main reason they were strongly encouraged by their administrators to join UA 
is because the school wanted more field trip vouchers. Meanwhile, teachers at another school are 
encouraged to take students on trips at least three times a year. Finally, one school is based on 
expeditionary learning so field trips and other out-of-school experiences are a core component of 
the school’s mission and approach. At these trip-heavy schools, it seems that vouchers make it 
easier for teachers to take their students on trips, but the vouchers do not necessarily induce 
teachers to take trips they would not otherwise have taken. 

At the schools without a school- wide focus on field trips, teachers exhibited considerable 
variation in their use of vouchers. Many teachers described their attempts to plan field trips that 
are integrated with other classroom work, but this is not always possible. At one school, the 
eighth-grade teacher typically uses vouchers as part of long-term science investigations, and 
students collect data or do other research that relates to their exit projects on class field trips. In 
the sixth and seventh grades at the same school, though, teachers use the vouchers “for trips to 
supplement the curriculum more.” 

Teacher voucher use also varied depending on teachers’ comfort with taking students on 
trips. At one school in Staten Island, one experienced UA teacher frequently takes class trips to 
the Bronx Zoo, while another teacher has never taken field trips because of safety concerns; her 
discomfort is influenced by the fact that she does not have tenure, that her principal encourages 
teachers to take all their students on trips (instead of just one class), and that there are not many 
science-rich cultural institutions located near her school. 

Field trips are inevitably complicated to plan, but some teachers use UA resources to help 
with the logistics. For example, there are designated staff members at each science-rich cultural 


IESP WORKING PAPER #01-14 


institution who help facilitate field trips, both with initial planning and with site-based tours and 
programming. One assistant principal explained that these resources have made it more feasible 
for parent coordinators and teachers to plan field trips for their students and families. At schools 
where teachers do not frequently take students on trips, teachers typically distribute vouchers to 
students to take individual trips with their families, and many award students extra credit for 
visiting science-rich cultural institutions on their own. 

Other Programs 

All schools in this study participate in other programs or special curricula in addition to 
UA. For example, one school offers two gifted programs as well as a magnet academy that 
emphasizes technology integration. Another school features a mentorship program where middle 
school students are matched with nearby elementary school students, and yet another school has 
several partnerships with community-based organizations and a school wide enrichment model 
(SEM) that emphasizes collaboration and student enrichment. As previously described, one 
school uses an expeditionary learning model, and it also is one of three schools around the city 
that supplements its math instruction with external teaching assistants. 

Three schools in particular are heavily involved with programs in addition to UA. At the 
first, the principal emphasized that he actively tries to get the school involved in as many 
programs as possible, as he wants to maximize the schools’ resources to engage and develop 
students. Related to science, this school has a cyber-based inquiry program, and it also 
participates in a pilot program that implements a comprehensive computer science/software 
engineering curriculum with the goal of helping students get into science, technology, 
engineering, and math (STEM) high schools and ultimately technology-based careers. Students 
also have the opportunity to participate in a robotics club and Science Olympiad. The school also 
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participates in a program with a nearby college, in which students travel to a college campus to 
use the facilities, listen to lectures by college professors, receive tutoring from college students, 
and apply to the college’s summer program. The school’s website also has information about a 
variety of other programs, such as the Let’s Move in School program, Family Fitness Club, 

Green Team, Math Counts, School Wellness Council, Saturday ELA program, book club, debate 
club, and more. 

The second school with notable programs is a selective school focused on math and 
science, which attracts and admits math- and science-minded students. A key component of the 
school’s founding mission statement is the curriculum, which integrates all subject areas in 
grade-level inquiry-based projects. The school sets aside about one day a week for students to 
work on these projects, and each unit includes a trip to a nearby institution. This school is also 
known for its specialized special education program for students with Autism Spectrum 
Disorders (ASDs), in which students with ASDs are integrated into regular education classes 
with targeted supports. Teachers also participate in a literacy program through a local university 
and an outside professional development program. 

The third program-heavy school is an all-girls school that has a special focus on engaging 
its students in science, technology, engineering, and math (STEM), as minority females are 
especially underrepresented in STEM fields. The school has strong supports around high school 
graduation and college-going through the Collegebound program. Additionally, several science 
teachers have been proactive in joining different programs and bringing them to the school. The 
school participates in a summer school field research program through a local university, which 
allows students to perform real-world field research with college students and faculty in the 
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summer and throughout the school year. The school also participates in a program that helps 
underrepresented schools prepare their students for competitive science competitions. 

VI. DISCUSSION 

In our quantitative analysis, we isolate the unique contribution of schools to science 
achievement, above and beyond school participation in UA and student characteristics. We find 
that UA schools do not, on average, have higher school effects than non-UA schools, suggesting 
that positive impacts of UA reflect program effects and not overall school quality. We use these 
school effects to identify UA schools that contribute positively to student science achievement 
and select schools for our case studies. 

Our qualitative analyses focused on the school-level characteristics that were unmeasured 
in our quantitative models and uncovered several themes that we believe are important for 
schools to successfully use the UA program to change teacher practice and improve the 
academic achievement of their students. These themes include school structure, the concentration 
of UA teachers in a school, teacher collaboration, administrative support around the program, 
resources and vouchers use, and involvement in other programs at a school. 

We find that school size matters, as does the concentration of UA teachers within a 
school. Teachers implementing UA in their classrooms have the greatest difficulty when there 
are relatively few science teachers in their school (e.g., one science teacher per grade) or when 
only a small number of science teachers in the school participate in UA. Although individual 
teachers in these cases may improve their classroom practices through UA, the influence of the 
program within the school is more limited than when several teachers are participating in the 
program. This is for two reasons: teacher collaboration and repeated student exposure to UA. 
First, teachers implementing UA in isolation expressed concern about developing ideas and 
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using tools, while UA teachers in schools with several teachers participating in the program 
appreciate having a more coherent science department. Second, in schools with a low 
concentration of UA teachers where students only have a teacher for one year, there may not be 
enough time for students to develop skills and build on UA concepts. 

We find that a culture of collaboration must exist within the school prior to adoption of 
the program. UA can strengthen this culture, but cannot build one if nothing exists at all prior to 
UA adoption. Collaboration among teachers in UA schools is an important aspect of program 
implementation. However, we find that UA does not create this culture within a school if it does 
not already exist. Rather, the program enhances the level of collaboration within schools. In 
other words, the UA program will not automatically create a positive culture or working 
environment for teachers and administrators in a school that is not functioning well. It can, 
however, enrich the quality of collaboration within schools. 

We find that successful UA schools are able to leverage multiple resources. That is, 
many schools that successfully implement UA are also implementing other programs within their 
schools, either science related or not, and are able to harness various resources to their 
advantage. There are no particular programs associated with facilitating the success of UA 
practice; rather successful schools are actively engaged in implementing other programs in 
addition to UA. For example, several schools have partnered with local universities for 
additional academic programs. Several of these programs are also related to science and provide 
additional resources such as college science facilities or technology. Other non-science programs 
offer resources such as teaching assistants for math classes or resources to promote healthy 
living. Other schools are involved with community-based organizations for enrichment 
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programs. Schools successfully implementing UA typically have experience working with other 
programs and understand how to maximize resources available to their schools. 

And finally, while not a school-level factor relating to the successful implementation of 
UA, we find that UA provides meaningful support for science teachers. The vast majority of 
teachers participating in UA say that it provides much-needed professional development for 
science teachers, who often do not receive science-specific support at their schools and who are 
increasingly asked to incorporate writing and critical thinking into their classrooms in order to 
help support ELA and math Common Core Learning Standards. The UA program’s professional 
development gives teachers resources and specific examples focused on how to incorporate 
rigorous, inquiry-based instruction into their classrooms, and many teachers in the UA program 
greatly appreciate U A as an enhancement to their teaching and believe the program has helped 
them meet the goals of the Common Core. UA promotes a hands-on, inquiry-driven approach to 
teaching understanding science concepts, and it provides a framework for evidence-based 
reasoning for scientific writing. UA teachers, who emphasize making and supporting evidence- 
based claims in science, provide educational experiences for their students that will serve them 
well in other courses as well, such as English language arts. 


VII. CONCLUSION 

This study joins a body of research that explores the role of informal science institutions 
in public science education. Based on the success of the Urban Advantage program in New York 
City, we conclude that informal science institutions can be important contributors to science 
learning in school. UA has been successful at improving the science achievement of middle 
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school students in NYC, but there is heterogeneity in its effect across schools. This heterogeneity 
is due in part to differences in program implementation. 

Our case studies revealed several school-level factors associated with successful 
programs. The most successful schools had the majority of science teachers participating in UA, 
ideally with teachers both within and across grades participating to maximize potential for 
horizontal and vertical curriculum alignment. Science departments were also more effective at 
implementing UA when they had common planning time and space to allow them to collaborate 
frequently. Additionally, schools were more successful when their administrators were 
supportive of science teachers in general, emphasized the UA program within the school, and 
provided the necessary structures to allow teachers to take students on class field trips (e.g., 
substitute teachers, chaperones). Finally, successful schools implemented other programs in 
conjunction with UA, as administrators and teachers in those schools were proactive in accessing 
resources and increasing opportunities for students and teachers. 

This study does have several limitations. First, because we are not able to match students 
to teachers due to data constraints, we are not able to define the UA treatment at the student level 
(based on a student actually having a UA teacher). Rather, we define UA at the school level, and 
all schools where at least one teacher is in UA are considered UA schools. This means some 
students are considered “treated” with UA even though they do not personally have a UA science 
teacher. This means our impact estimates are biased downward, and the true “impact” of UA is 
likely larger than we have estimated. Another key limitation is that like all qualitative studies, 
our case studies do not provide causal estimates. For example, we cannot definitely conclude that 
collaboration causes success with UA or that increasing the concentration of UA teachers at a 
school would increase science achievement. Rather, we have identified school factors that are 


IESP WORKING PAPER #01-14 


associated with successful implementation of the program. And finally, as is common in many 
qualitative studies, we were not able to recruit a sufficient number of low performing schools. 
Therefore, we were not able to accurately assess whether all of the factors associated with higher 
performing UA schools were or were not occurring in these schools. 
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Table 1: Impact of UA on 8 th -Grade Science Achievement, 2004-2010 


Outcome: Z-science 

(1) 

Year Before Joining UA 

First Year in UA 

Years Post UA 

Black 

Hispanic 

Asian 

Female 

LEP 

Special Education 

Free or reduced-price lunch 

Constant 

0.011 

(0.021) 

0.044* 

(0.024) 

0.056** 

(0.028) 

-0.397*** 

(0.017) 

-0.226*** 

(0.015) 

0.162*** 

(0.019) 

-0.072*** 

(0.005) 

-0.811*** 

(0.026) 

-0.592*** 

(0.011) 

-0.092*** 

(0.008) 

45.914*** 

(8.097) 

Year fixed effects 
School fixed effects 
Observations 
R-squared 
Number of Schools 

Yes 

Yes 

400,872 

0.355 

519 


Robust standard errors in parentheses 
*** p<0.01, ** p<0.05, * p<0.1 
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Table 2: Distribution of school effects by UA status 


Mean 

Min 

Max 

Std. Dev. 


Never UA 

0.007 

-0.967 

1.394 

0.441 


UA for at 

least 2 yrs 

-0.027 

-0.803 

1.047 

0.342 


UA for at 

least 4 yrs 

-0.014 

-0.491 

1.047 

0.359 


Table 3: UA Schools with the highest and lowest school effects, UA for 2+ years 


Highest school effects 


Number of 


School Name 

School Effect 

Years in UA 

Center School 

1.047 

5 

Treetop School 

0.971 

5 

Tower Academy 

0.937 

3 

Easton School 

0.758 

2 

Parkview Intermediate 

0.649 

3 

Galileo School 

0.604 

4 

Anton Academy 

0.602 

4 

Northside Middle School 

0.585 

3 

River Academy 

0.514 

2 

Edgeview Middle School 

0.509 

2 

Lowest school effects 


Number of 

School Name 

School Effect 

Years in UA 

Martin Middle 

-0.482 

5 

Southside School 

-0.491 

5 

Pleasant Ridge 

-0.496 

2 

Smith School 

-0.497 

2 

Jefferson Intermediate School 

-0.499 

2 

Flagstaff Intermediate 

-0.524 

2 

Howard School 

-0.549 

3 

Riverbank Prep 

-0.572 

2 

Ashton Prep 

-0.598 

2 

Cedarwood School 

-0.803 

2 
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Table 4: Demographic characteristics and academic performance of case study schools 


School Name 

Empower 

Academy 

Tighe 

J.H.S 

Hilltop 

School 

Galileo 

School 

Fiske 

J.H.S 

Moray 

M.S. 

Salem 

J.H.S. 

Anderson 

I.S. 

Mill Creek 
I.S. 











School effect 

0.338 

-0.190 

-0.203 

0.604 

0.298 

0.381 

0.400 

0.192 

0.277 

Borough 

M 

M 

M 

K 

K 

Q 

Q 

R 

R 

Years UA 

3 

5 

3 

4 

5 

2 

4 

3 

4 

% White 

2% 

1% 

0% 

40% 

40% 

38% 

21% 

83% 

89% 

% Black 

31% 

7% 

2% 

27% 

19% 

6% 

5% 

2% 

0% 

% Hispanic 

60% 

92% 

97% 

21% 

13% 

23% 

9% 

11% 

9% 

% Asian 

5% 

0% 

0% 

11% 

28% 

33% 

64% 

4% 

2% 

% Female 

100% 

48% 

44% 

45% 

50% 

51% 

49% 

45% 

49% 

% Free/Reduced Lunch 

80% 

97% 

92% 

31% 

68% 

51% 

49% 

39% 

34% 

% ELL 

2% 

38% 

28% 

1% 

7% 

6% 

5% 

2% 

1% 

% SPED 

5% 

17% 

20% 

21% 

7% 

17% 

12% 

17% 

19% 

Avg Z-Math 

0.31 

-0.44 

0.05 

0.73 

0.78 

0.62 

0.99 

0.24 

0.47 

Avg Z-ELA 

0.37 

-0.48 

-0.2 

0.83 

0.61 

0.55 

0.82 

0.34 

0.52 

Avg Z-Sci 

0.53 

-0.29 

-0.39 

0.93 

0.6 

0.84 

0.5 

0.32 

0.48 

% Prof. Math 

77.8% 

39.7% 

68.7% 

92.8% 

88.2% 

84.3% 

92.7% 

72.2% 

82.9% 

% Prof. ELA 

60.8% 

25.0% 

29.0% 

82.9% 

72.6% 

63.7% 

77.5% 

57.1% 

66.7% 

% Prof. Science 

78.6% 

43.6% 

35.5% 

91.8% 

79.7% 

95.9% 

80.8% 

69.8% 

78.7% 


M = Manhattan; K = Brooklyn; Q = Queens, R = Staten Island 
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APPENDIX A: TEACHER INTERVIEW PROTOCOL 

Teacher Background 

• What do you teach? 

• How long have you been teaching? At this school? 

• How long have you been involved with UA? 

• How did you become involved with UA? Why did you become involved with UA? 

• How many science teachers do you currently have? 

Teaching Practices 

• What is your approach to teaching science? 

If teacher is in UA 

• Do you use the IDD/DSET on a regular basis or just for specific lessons? 

• How are the tools introduced - are they introduced all at once or scaffolded? 

• Have you modified the tools in any way to make them more useful? 

• What is the level of collaboration and awareness of these tools among UA and non-UA 
teachers across the science department? Across grades? 

• How frequently are UA concepts and strategies (claims, evidence, reasoning, relationship 
between independent/dependent variables) used in lessons? 

• How do you make use of the vouchers and the availability of the partners? 

• When do students start working on exit projects? How do they use the partners in 
developing them/collecting data, etc? 

• Can you think of ways in which your practice might have changed since you became part 
of the UA project? (Inside or outside the classroom) 

If teacher is not in UA 

• How much do you know about UA? 

• Do you ever collaborate with UA teachers? 

• What do you think of UA? Or why have you decided not to participate? 

Science at the School Level 

• Does the school have a science leadership team? Grade level science meetings? Science 
department meetings? 

• Does the school have a shared mission and vision about science? 

• What is the level of science teacher turnover in this school? 

• How is achievement data used to improve instruction? 

UA at the School Level - to be asked of UA teachers 

• What is the role of the lead teacher within your school? 

• How involved are families in the program? Does the parent coordinator work with them 
to do Lamily Science Night or take them to the museums/zoos/gardens? 

• How do you use the materials that are purchased with UA funds? Do non-UA teachers 
have access to them also? 

• How well do the administrators at the school support UA? In what ways? 

Anything else you would like to share? 
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